Skip to content

Skip exact COUNT(*) for large tables, use database statistics estimate#639

Merged
datlechin merged 3 commits intomainfrom
perf/skip-count-large-tables
Apr 8, 2026
Merged

Skip exact COUNT(*) for large tables, use database statistics estimate#639
datlechin merged 3 commits intomainfrom
perf/skip-count-large-tables

Conversation

@datlechin
Copy link
Copy Markdown
Collaborator

Summary

Fixes the 10-20s loading delay on large PostgreSQL tables reported in #519.

Root cause: Phase 2 automatically runs SELECT COUNT(*) FROM table on every table open. PostgreSQL COUNT(*) requires a full sequential scan (MVCC — each row's visibility must be checked). On a 3M-row table this takes 10-20+ seconds and blocks the shared driver connection.

Fix: Skip exact COUNT(*) when the approximate count (from pg_class.reltuples / information_schema.TABLE_ROWS) exceeds a configurable threshold. This is the industry-standard approach:

Tool Approach
TablePlus "Estimate count when table has more than X rows" setting
pgAdmin "Count rows if estimated less than 2,000" (default)
DBeaver Does NOT count automatically — manual button
DataGrip Does NOT count on table open

Changes

  • Add countRowsIfEstimateLessThan setting (default: 100,000)
  • Skip exact COUNT in both launchPhase2Work and launchPhase2Count when estimate >= threshold
  • Add picker in Settings > Data Grid > Pagination (1K / 10K / 100K / 1M / Always count)
  • Tables above threshold show ~N rows using the ~ prefix already supported by the status bar

Before/After

  • Before: Open 3M-row table → wait 20s for COUNT(*) → "1-1000 of 3,000,000 rows"
  • After: Open 3M-row table → instant → "1-1000 of ~3,000,000 rows"

Partially addresses #519

Test plan

  • Open a large table (>100K rows) — should show ~N rows instantly, no delay
  • Open a small table (<100K rows) — should still show exact count after Phase 2
  • Settings > Data Grid > "Count rows if estimate less than" — verify picker works
  • Set to "Always count" — verify exact COUNT runs for all tables
  • Check non-SQL databases (Redis) — should continue using approximate counts

@datlechin datlechin merged commit 60e812a into main Apr 8, 2026
2 checks passed
@datlechin datlechin deleted the perf/skip-count-large-tables branch April 8, 2026 06:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant